---
sidebar_position: 2
---

import useBaseUrl from "@docusaurus/useBaseUrl";

# Ingestion and Sources

Log events are ingested into Logflare through the **ingestion API**, where JSON payloads are sent to Logflare to be processed and inserted into the configured backend.

These processed payloads will be inserted into **Sources**. A **Source** is made of up of many log events. Log events may have one or more keys, but will always have 3 standard top-level keys:

1. `id`
2. `event_message`
3. `metadata`

The Logflare service will insert into Logflare-managed BigQuery for users on the Free and Metered plan.

## Creating a Source

Before you can ingest logs, you need to create a source:

1. Log into your [Logflare account](https://logflare.app/)
2. Click the **"New Source"** button on the main dashboard
3. Enter a unique name for your source
4. Click **"Create"**

After creating a source, click the **Setup** button to retrieve your **Source ID** (UUID), and navigate to **access tokens** to generate an ingestion key - you'll need these credentials to send log events.

### Important: Configure Key Settings After Creation

When you first create a source, it uses default settings that **may not be suitable for all use cases**. We strongly recommend reviewing these settings immediately:

#### Backend TTL (Data Retention)

The default TTL is typically **7 days on free plans**, which may be too short for audit logs or compliance requirements.

**To configure**: Navigate to Source > Edit > Backend TTL

:::warning Configure TTL Before Ingesting Production Data
Once data exceeds the TTL period, it is **permanently deleted** and cannot be recovered. Always set an appropriate retention period before sending production logs, especially for audit trails.
:::

See [Backend TTL](#backend-time-to-live-ttl) below for more details.

#### Schema Lock

By default, Logflare automatically detects and adds new fields from incoming events ([Adaptive Schema](#adaptive-schema)). Once your schema is stable, consider **locking it** to prevent unwanted changes.

**To configure**: Navigate to Source > Edit > Lock Schema

#### Other Important Settings

- **Suggested Search Keys**: Define fields that should be included in queries for optimal performance (useful with BigQuery clustering)
- **Notifications**: Set up alerts for schema changes, errors, or specific log patterns

All source settings can be accessed by clicking the **Edit** or **Settings** button on your source page.

## API Endpoints

There are two ways in which you can ingest data into Logflare, via specifying the source UUID, or via the source's name.

```
POST https://api.logflare.app/api/logs?source=9dd9a6f6-8e9b-4fa4-b682-4f2f5cd99da3

POST https://api.logflare.app/api/logs?source_name=my.logs.source
```

OpenAPI documentation for ingestion can be found [here](https://logflare.app/swaggerui#/Public).

### Batching Your Events

You can ingest events individually, or via a batch.

To ingest individually (not recommended, as it is slower), send your request with the following JSON body:

```json
{
  "message": "your log event message",
  "metadata": {...}
}
```

To ingest by batch, send your request with the following JSON body:

```json
{
  "batch": [
    {"message": "your event message 1", "metadata": {...}},
    {"message": "your event message 2", "metadata": {...}},
    ...
  ]
}
```

Note that if you have mutliple sources with the same name, it will result in an error on ingestion and the log event will be discarded.

## Pipeline Transformations

At ingestion, the pipeline will perform the following in sequence:

1. Add in the `id`, `timestamp`, and `event_message` fields.

- `event_message` will be populated from the special `message` field if missing.

2. Ensure that all field names adhere to the BigQuery column requirements. See [key transformation](#key-transformation) for more details.

- Any fields that are automatically adjusted will be prefixed with an underscore (`_`).

3. If set, fields will be copied in sequence. See [field copying](#copy-fields) for more details.

### Key Transformation

When logging object, your object keys will be transformed automatically to comply with the respective backend in use. For example, BigQuery column requirements require that names only contain letters (a-z, A-Z), numbers (0-9), or underscores (\_), and it must start with a letter or underscore. This will be automatically handled for you when ingesting data.

### Copy Fields

A source can be configured to copy fields from one path to another. This allows for augmentation of the event at ingestion time before inserting into the underlying backend-ttl

A `:` symbol is used as a delimiter between the source field and destination field. The pattern is `source:destination`. Dot syntax is used for specifying nested paths.

For example, to copy a field from a nested field to the top-level (for performance reasons or otherwise)

```text
metadata.nested_field:top_field
```

Multiple rules can be specified and chained, as they are executed in sequence.

```text
metadata.my_nested.field:top
top:top_level_copy
top:backup_copy
```

In this example, 3 additional fields will be created: `top`, `top_level_copy`, and `backup_copy`.

As field name transformations to BigQuery specification occur before this step, modified field names must be used.

For example, if a payload with dashes is ingested:

```json
{ "top-level": 123, ...}
```

The field will be converted into `_top_level` in the field name transformation step. We will then have to refer to it as such in the Copy fields configuration:

```text
_top_level:my_copied_field
```

:::note
Destination field names must match the BigQuery colummn name specification, or the event risks being rejected.
:::

## Adaptive Schema

As your logging needs changes, Logflare is capable of detecting and adjusting the database schema accordingly. This allows you to focus on analyzing your logs instead of having to manage your logging pipeline manually.

Suppose your initial log events had the following shape:

```json
{
  "message": "This is my log event",
  "metadata": {
    "my": "first log"
  }
}
```

The generated schema would be the following:

```
message: string;
metadata: {
    my: string;
}
```

As your application requirements change, suppose you now need to add in new information to your log events.

```json
{
  "message": "This is my new log event",
  "metadata": {
    "my": "first log",
    "counter": 123
  }
}
```

Logflare will now detect the schema changes in the event, and add in the new column to the source's underlying table.

```ts
message: string;
metadata: {
  my: string;
  counter: number;
}
```

:::note
On high ingestion volume, Logflare will sample incoming events instead of checking each event. The sample rate decreases as the ingestion rate increases. Ingestion rates are compared only on an individual local server that is performing the ingestion.

The sample probability is dynamically calculated as `1.0 / average_rate` with safety bounds (minimum 0.00001, maximum 1.0), which maintains approximately 1 schema update per second across all rate levels. This supports rates up to 100,000+ events per second.
:::

### Schema Changes

The schema changes is done automatically. If this is not the desired behaviour, you can disable this by locking the schema in the source's settings.

#### Additive Changes Only

Schema updates are additive and new columns cannot be removed through the UI. We recommend creating a new source if a new schema is desired, and migrating any existing data manually.

### Backend Time-To-Live (TTL)

Backend TTL refers to how long data is retained within a given backend. Logflare will manage the retention settings for each backend where available.

:::warning Use Logflare for Retention Handling Only
It is not advisable to manage the backend's retention settings directly without going through Logflare, as conflicts may arise.
:::

You can configure the Backend TTL via source settings under Source > Edit > Backend TTL.

![Backend TTL Setting](./backend-ttl.png)

Please refer to backend-specific documentation for in-depth explanation for how retention is handled.

## Querying

Logflare provides a filtering UI to expressively query for log events. Currently, the [Logflare Query Language](/concepts/lql) is supported.

<img
  alt="Search UI"
  src={require("./search-ui.png").default}
  style={{ maxHeight: 800, maxWidth: "100%" }}
/>

Most of the time, you would be making queries to sources for certain keywords inside
of the event message. In most cases, one can rely on using simple text search by
entering the text directly into the search bar without any special syntax.

To search for specific phrases, we can wrap the query with double quotes, like this `"my text"`.

All keyword searches are case sensitive.

### Source Schemas

To quickly view all available source fields that can be filtered on, the source's schema reference is provided within the search UI. It can be accessed via the **schema** source navigation button.

![Source Schema Reference](./schema-reference.png)

### Live Search

When searching logs, you can turn your current search into a **Live Search**, which tails the source and streams new events that match the filter into the search results.

### Query Saving

Queries that are frequently used can be saved, and will be available for quick access from the dashboard.

### Suggested Search Keys

To encourage usage of a particular search key filter, configure the **Suggested Search Keys** setting to trigger a check on each search query made through the UI.

This is particularly useful when there are custom partitions set up for certain columns in the underlying backend table.

For example, if there is an existing optimization made for the string column called `level`, we include it into the suggested search keys. Users will then be prompted to include the `level` filter in their LQL search queries, and will be given the option to force the query as well.

### Querying Outside of Logflare

Certain backends provide querying functionality. Documentation on each can be found in specific backend guide.

- [Querying in BigQuery](/backends/bigquery#querying)

## Rules

Source rules specify additional event routing conditions when incoming log events are received. Log events sent to a source that meet the conditions of a rule will also be sent to the source specified by the rule.

Source rules can be accessed by navigating to the source page and clicking on the **rules** navigation item.

![Navigate to the rules section](./rules-navigate.png)

Source rules accept an [LQL expression](/concepts/lql) as well as a target **sink source** to route events to.

![Source Rules](./source-rules.png)

Events can only be copied once, in order to prevent infinite copying and redundancies.

:::info Regex Rules Deprecation Notice
Regex rules are officially deprecated and will be removed in future versions. All new rules created should only be in LQL.

Regex conditions can be achieved with [LQL through the `~` operator.](/concepts/lql#metadata-filtering)
:::

## Ingestion Parsers

Logflare features ingestion parsers for different types of event formats. These out-of-the-box processors will help to auto-generate certain fields or reformat the payload for storage.

### Cloud Event

The supported spec is [v1.0.2](https://github.com/cloudevents/spec/blob/v1.0.2/cloudevents/spec.md).

#### Example

```
POST https://api.logflare.app/api/logs/cloud-event?source=f6cccd3a-c42e-40f7-9b01-95d3699d3113
Content-Type: application/json
X-API-KEY: XXXXX
CE-specversion: 1.0
CE-id: 01HPPC9X0HPKB8E1RSPA5YFZB2
CE-source: flyd
CE-type: io.fly.machine.start
CE-time: 2024-02-15T12:36:45+00:00

{
  "body": {},
  "machine_id": "148ed193b95948",
  "status": "started"
}
```

The corresponding event will be processed and ingested, where ellipsis represent autogenerated fields:

```json
{
  "cloud_event": {
    "specversion": "1.0",
    "id": "01HPPC9X0HPKB8E1RSPA5YFZB2",
    "source": "flyd",
    "type": "io.fly.machine.start",
    "time": "2024-02-15T12:36:45+00:00"
  },
  "machine_id": "148ed193b95948",
  "status": "started",
  "event_message": "...",
  "timestamp": "...",
  "id": "..."
}
```

## Cloudflare Logpush

Logflare supports ingestion of logs from Cloudflare through the [**Logpush API**](https://developers.cloudflare.com/logs/about/). This allows you to send logs directly from Cloudflare to Logflare for processing and storage.

To ingest logs from Cloudflare, use the following configuration for the [HTTP Destination](https://developers.cloudflare.com/logs/get-started/enable-destinations/http/):

```text
https://api.logflare.app/api/logs/cloudflare?source=f6cccd3a-c42e-40f7-9b01-95d3699d3113&header_X_API_KEY=xxxxxxx
```

Replace the source UUID and the api key value with your own source UUID and [access token](/concepts/access-tokens/). We recommend using a source-scoped ingest access token for better security.

The configuration is identical for both Logpush and Edge Log Delivery.

## Ingestion Validation

Logflare will reject certain payloads containing values of certain shapes:

1. Arrays of Arrays - `[["testing]]`
2. Mixed Arrays - `["some", {"obj": 123}]`
